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Most of you are familiar with the time-honored way of developing a cur- 
riculum, After deciding what your topic Is, you might sit In front of a 
blank piece of paper and write lectures and a topical outline. You probably 
would refer to the best authorities In the field, and you would arrange source 
material and readings for the students. When all Is In readiness, you would 
expose the curriculum to students and hope for the best. 

nowadays, there Is much talk about specifying objectives before designing 
the curriculum. Some well-known designers of instruction go so far as to 
recommend writing test items or criterion exams before beginning Instruction! 
This climate makes It hard to expose oneself to the world as a traditional 
curriculum writer, but there are very few of us v/ho believe It is practical 



actually to work in this new way. 

The message of this talk Is that we have found it practical to work In 
this way and we recommend it. 

First of all, we were convinced by the reasonable argument that the tra- 
ditional approach to curriculum development was actually a method for Isolat- 
ing the developer and the curriculum from important Influences. If he has 
no feedback from the final audience for his work, he does not know If it Is 
successful. The final audience Is "out there" beyond the classroom or school. 
Information on our effects beyond the classroom Is needed in order to help us 
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decide 'vhat to teach. Determination of effective goals requires a reference 
point beyond the final examination. 

The traditional curriculum approach also cuts the teacher off from student 
response until the curriculum Is finished. At that point when students have 
trouble with the curriculum In the classroom, It Is often too late to revise. 

We were convinced that feedback was needed at aJQ. stages of curriculum 
development so that we could learn from our errors and make Improvements. We 
were concerned with errors of two kinds: errors In choice of objectives and 
errors In method of Instruction, 

First we set our final goals and decided on the measures which would be 
our signs of success. One goal was to be so useful to schools that they would 
adopt our training and its products. The measure of this would be the behav- 
ior of educators who decide on school policy. If we were successful, they 
would find the skills we were teaching to be worth budget allotments and would 
create positions for people with these skills. 

The next kind of success we hoped for was Improvement In the complete 
range of foreign language skills thought to be Important by content specialists. 
This effect will be measured by comparisons of gain scores betv/een treatment 
schools and ccntrol schools, using student perfomance on the Modern Language 
Association Cooperative test of foreign language skills. 

The most important outcome to us would be the successful achievement of 
many goals set by teachers for their students. To measure this we are keeping 
records of each problem and Its solution In terms of student performance and 
teacher satisfaction. 

Having stated our ultimate goals and having been explicit about how we 
would recognize them, we had to state the objectives of the curiiculum that we 



. 3 - 

thought would train people to reach those goals. This process was empirical 
and has Just been described briefly by Roger Scott. The decisions and actions 
in the problem-solving routine he discussed constitute the objectives for our 
curriculum. We tried to specify as many as possible of the sub-skills that 
composed the parts of the problem-solving routine. This resulted i., written 
descriptions of approximately 100 skills. 

Since these skills were highly interrelated, we grouped them into six 
reasonable areas. These are listed on the first page of the handout you have 
Just received. (Three units are described in some detail on the following 
four pages.) They represent the main stages of the problem-solving routine: 
Problem Explication includes interviewing the teacher to choose a problem and 
state it in explicit terms. Quantification of Behavior includes many skills 
of translating teacher objectives into valid and reliable measures of student 
behavior. Behavior Modification and Instructional Design are the two main 
strategies for analyzing and solving classroom problems. Research involves 
skills in generating, conducting and reporting experimental studies. Dissemi- 
nation and Maintenance includes analysis of the school as a system and the use 
of available and created resources, in order to spread and establish success- 
ful innovations. 

It was at this point that two unusuai things happened. First of all, we 
remembered that we had endorsed the "new look" in curriculum development - 
specifying objectives and writing criterion tests before developing instruc- 
tion - and having remembered, we actually stuck to our resolve. We proceeded 
to write as many test items as we could in each of the six skill areas. 

From these v/e put together a pre-test vfhich covered the range of ISC 
skills. Some samples of questions in that pre-test appear on pages 6-12 at 
the end of the Appendix. In order to write these and the other test items, we 




drew upon hundreds of problems which we had collected previously in talks with 
teachers. 

The first pre-test item in the handout asks the trainee to identify 
whether responses in a transcript of an actual interview helped or hindered 
progress tov/ard a clear statement of a teacher's problem. In the second test 
item, the trainee is given corrected answer sheets, graph paper and a teacher's 
objective; he must report to the teacher on the success of an attempted solu- 
t^on. The third item presents examples of ISC comments during the trial of 
some teaching material. He must choosf the more appropriate response in each 
instance. These sample, respectively, the curriculum areas of Problem Explica- 
tion, Quantification of Behavior and Dissemination, and Instructional Design. 

We administered the complete pre-test (which runs to about 30 pages) to 
our trainees, to ourselves and to a control group of advanced students who were 
taking a School of Educa'don course in Modern High School Curriculum. With a 
parallet test in hand to use as a post- test, we were flushed vdth righteousness. 
Again, we were tempted to teach. 

However, the pre-test proved to be far from exhaustive. Therefore, we 
still needed means to knov/ if we were teaching the sub-skills which composed the 
six areas of instruction. 

We wrote more test items. As of today, we have written hundreds of items 
and expect to write more. 

I will return to the hundreds of test items and their impact on the cur- 
riculum, First I want to tell you about the other unusual happening- 

The second unusual happening was our choice of a training and feedback 
strategy. Ue made tne first trainees Integral parts of the curriculum develop- 
ment process and established feedback channels both from our efforts at train- 
ing and from the effects of our trainees upon teachers and students in real 
schools. 
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We decided that whatever we taught, we wanted to know immediately whe*’'*er 
the skills involved were working in the school T1& “this eyJi, ^ 

hired two trainees at a time when we only had test items written in our cur- 
riculum. One trainee had many years of teaching experience. She has corrected 
many of our errors before they reached the schools and has provided many of the 
problem descriptions which were developed into test items. The other trainee 
had extensive experience in the use of behavior modification techniques with 
children. He was able to write many of the test items in his area of speciali- 
zation and give us pointers on consulting with teachers. 

With little further ado, we sent these brave souls into the field. We 
sought and still seek teachers who, attracted by our glowing descriptions of 
ISC virtues, would allow the staff and trainees to interview them. We arranged 
to record and transcribe the interviews. These interviews attempt to get a 
teacher to state an instructional problem she has. Each teacher problem goes 
through role-playing by the staff during a weekly class session. In addition, 
the trainees keep a daily log of all their activities and impressions of both 
training and field experience. 

With all the feedback provided by these arrangements, with the two first 
trainees and the hundreds of test items, we feel satisfied that we have not 
isolated ourselves as curriculum developers in the ways mentioned at the start 
of this talk. In fact, the whole project is as bugged as a foreign embassy. 

What is the result of this extreme emphasis on feedback and measurement? 
The benefits fall under two headings: (1) empirical validation of curriculum 

content and means of instruction; (2) efficiency in Instructional design. 

Our content is vallaated by actual use in the kinds of schools we hope to 
affect, and it is revised as needed. The means of instruction are validated 
because we revise them until they work. 
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Our process of Instructional designing Is efficient In the following ways: 

1. Pre-tests tell us what skills are needed by each trainee* They need 
learn only what they do not know. 

2. Teaching Is simplified. Our many test Items, when sequenced, provide 
such clear directions to trainees that merely attempting an answer 
and then finding the right answer Is often all the Instruction needed. 

3. We, as designers of the curriculum, are presented with a means for 
selecting materials which Is organically bound to our purposes. The 
test Items are clear-cut guides to choosing chapters In books, pro- 
grammed texts, films, tapes, etc. Now that we are getting to the 
teaching, we are finding It very easy to pick materials to fit the 
test Items. We can easily and quickly tell how effective each bit 
of Instruction Is. And most Importantly, we are more confident than 
we have ever been In our teaching careers that these learning experi- 
ences are clearly relevant. 
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Appendix to accompany 



"Training the Innovative Agent** 




Appendix 



The Training Curriculum 



♦Unit I: 
♦Unit II: 
♦Unit III: 
Unit IV: 
Unit V: 
Unit VI: 



Problem Explication 
Quantification of Behavior 
Behavior Modification 
Instructional Design 
Research 

Discrimination and Maintenance 



♦These units are described in more detail on the following pages 



T . Problem Explication 

Much of the ISC*s time will be spent in informal interviews 
with teachers. 

Faced with the complexities of the classroom^ teachers can 
be discouraged from trying new strategies; their close involvement 
may hinder finding a solution. 

The ISC will help the teacher see problems in a new light: 
he assures her that help is available to carry out a solution, 
he encourages trying a new approach, but he first starts her 
toward an objective description of the problem. 

Working on her own, a teacher often describes problems in 
terms of variables she cannot control such a.« I.Q., poor home 
environment, etc. She is also tempted to move quickly to familiar 
or traditional solutions. 

Trainees will be able to question the teacher so that a 
catalog of variables, relevant to problem solution evolves. He 
will suspend judgment on solutions until sufficient facts are in. 
He will encourage her to originate rather than adopt traditional 
solutions. 
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Appendix 



T^en a teacher tries a new strategy with her students# she 
frequently lacks clear signs of success. 

The Consultant will interview the teacher to obtain the 
outlines of pre-test procedures and will set up clear-cut measures 
for recognizing success. 

The ISC will have the component skills of interviewing' and 
following up an interview. He will be able to: 

— choose a good interview when it is compared V7ith an 
inadequate interview. 

— choose the best approximation of a good interview 
when given several choices. 

— ask efficiently the questions which help a teacher 
produce needed information. 

— maintain teacher interest and involvement during an 
interview. 

— provide next steps for a teacher to work on after 
an interview. 

— write a brief memo after each interview which: 

shows a teacher how her own statement of a 
problem can be restated in behavioral terms 

assures the teacher that the ISC has listened 
carefully 

provides a further stimulus for the teacher 
to collect data as the first step in solving 
a problem. 
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Quantification of Behavior Unit 



Problem 

Teachers are often stopped 
by the complexity of problems 
and fail to reduce them to 
workable parts. 

Sometimes teachers are criti- 
cized for teaching toward 
controversial, harmful or 
useless goals. 

The mass of test data collected 
in schools can easily overwhelm 
a teacher. 

Sometimes a teacher needs a 
test or measure of behavior 
that is not commercially 
available. 

Failure to obtain frequent and 
useful measures of student 
performance is often related 
to the teacher's lack of time. 



Teachers and test experts are 
often at odds over acceptable 
tests and testir^g *"trategies. 



Teachers often find it hard 
to interpret and present teat 
data. 



Relevant ISC skill 

The ISC v/ill learn to do task 
analysis. 



The ISC will learn to aid teachers 
in establishing objectives 



The ISC will help the teacher 
choose proper uses of test data 
that have already been collected. 

The ISC will assist teachers in 
building their own tests and 
behavioral measures. 



The ISC will be able to assist 
her in obtaining and interpreting 
the data, and show her a variety 
of techviiques which students can 
do themselves. 

The ISC will design strategies 
fcr testing and will select and 
construct tests according to 
criteria acceptable to measure- 
ment experts and to teachers. 

The ISC will interpret and 
piesent test data without dis- 
tortion and in ways useful to 
teachers and situdonts. 
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Behavior Modification Unit 



Problem 

Teachers often describe 
students and their problems 
in terms of inferences which 
hinder quantification* mani- 
pulation* and evaluation. 

Identifying effective rein- 
forcers is often a difficult 
task for teachers 



Teachers may unwittingly 
maintain undesired behavior 
in students or fail to set up 
appropriate contingencies for 
desired behavior. 

Often, teachers do not know 
effective ways of monitoring 
student oehavior. 



Solutions for classroom . 
problems — which appear theor- 
etically sound — sometimes pro- 
duce aversive consequences for 
the teacher when implemented 

Educators in the applied set- 
ting often become isolated 
from the stimulation and the 
information offered in the 
research setting. 



Solutions proposed for educa- 
tional problems are often too 
cumbersome for the system and 
are dropped when the consultant 
leaves . 



Relevant ISC Skill 

The ISC will learn to translate 
teachers' statements into opera- 
tional language. 



Trainees will learn techniques 
of identifying available and 
potential reinforcers for 
education 

The Consultant will be trained 
in contingency management. 



The ISC will learn a variety of 
techniques that will aid the 
teacher in obtaining more exact 
and more frequent information 
about her students. 

To insure successful solutions, 
the ISC will be trained to con- 
sider reinforcing aspects for 
both teachers and students. 



The trainee will have skills in 
research design, and the experi- 
mental analysis of behavior* 
which will facilitate a continu- 
ing contact with the research 
community. 

The Instructional Systems Con- 
sultant will learn techniques of 
maintaining behavioral change 
that are "tailor made" for the 
particular school system. 
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On the follov^/ing pages you will find a portion of an interview 
between a teacher and a consultant. 

The consultant's major objective is to discover the exact m ture 
of the teacher's problem - to be exact enough that the worth of the 
proposed solution can be accurately determined. 

For each of the consultant's statements in the interview, place a 
check in the box marked with a plus [ □ +] if the statement definitely 
led toward the above objective. Otherwise, place a check in the box 

B 

marked minus CD -3. 



TEACHER-COriSULTAi'ff INTERVIEW 

Teacher: Next week I'm going to be presenting a unit in my first 

year French class on Impressionist painters, and I was 
told that there are several good art films available, on 
French art - maybe on France, some of the museums. I 
thought that maybe you could help me obtain a film so that 
, when we do the unit I would have this to shov/ to them. 



Consultant: I have several catalogs of films that we could go through 
that might help. Right now you should tell me just what 
types of things you would like in the film. What things 
do you think it is important for the children to see? 

□ + □- 

T: Well, I think it would be important for them to get the 
background of a good deal of the cultural content of the 
artists of the period, and preferably in French; showing 
some of the works of the artists with a great deal of cul- 
tural commentary in it. Othemse, looking at the paint- 
ings wouldn't be of much value. But if they understand 
the artist or tct^nthing about him, his part in Paris or 
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the part he played in French literature of the period, 
this could be Important - not just looking at films and 
being bored besides. 

You want a lot of things about each artist. In terms of... 

□+ □- 

...the period In which he lived, where he lived In Paris, 
did he paint In the provinces of France, what major works of 
his are well known, can you see them In the museums of 
Paris, or can you go to see them somewhere else in France? 

And you are Interested particularly In the Impressionist 
painters, like... 

a* p- 

Renoir, «.onet - the v/hole school and the Post-Impression- 
ists. The whole unit Is on Monet, Manet, Degas, Renoir. 

Yes, certainly the paintings of Manet should he seen and 
appreciated by every French Student. 

n* □- 

Right! And that's not the case now. 

Suppose you *'ere tryinn to make a test for the kids after 
they'd seen this film to see If they had been helped In 
any way. What kinds of things would you ask? Maybe you 

Include something like this In the present test that you 
have. 
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T: Weil, to be truthful, I vvant to use the film as a kind of 
embroidery on the grammar unit that Is being presented. 

I don*t know that I would want to greatly stress the con- 
tent of the film Itself. 

It will have a conversation in French about going to the 
museum and seeing the Impressionists, “Who are the Impres- 
sionists?" and so on would all be In French. I would want 
them to understand the structures In the unit In such a way 
that they're not talking In the abstract about something 
that they know nothing about. I'd like the film to make 
the unit concrete. 

C: What Is the name of the unit? 

□+ □- 

T: “The Impressionists," 

C: Then one of the reasons you're studying the unit Is to 
Increase their skill In grammar and vocabulary Items? 

□+ D- 

T: Yes, certainly. If It's possible - In time - and If they 
can absorb that much - they could learn something about 
the Impressionists. Then I can hold them responsible for 
that. It would be difficult to test an art unit, because 
I don't vvant to just shoi/ them a painting and ask whose 
It Is - If It's something well known, by Van Gogh or 
Renoir or someone like this. It Is so obvious. They could 
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tell the difference between a Renoir and a Van Gogh. I 
might ask them to identify the painter, but not necessarily 
the title of the work of art. General questions like, 

“Uhen did Van Gogh live?" They would learn this from the 
film. Also, "i^ame three Impressionist painters," and 
"What museum vjould you go to in Paris to see Impressionist 
works?" 

C: So you sort of have a dual puroose involved. One is the 
grammar and vocabulary items included in the unit and the 
second is supplementary learning about the French culture 
and painting. Is that a fair statement? 

□ + □- 

T: Yes, that's correct. That's exactly what it is. 
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A French teacher has, as an instructional objective, that when 
given the infinitive of the 10 irregular verbs listed on the enclosed 
test, the student will correctly write 8 of the 10 in tl:e past tense. 
There are five students in her class. 

a) Using the data from the attached test set "A", prepare a graphi- 
cal presentation of the pre- and post- test scores. {Use the graph 
paper on the next page.) 

b) Write a memo tu the teacher explaining the success of the instruc- 
tion in this area and suggesting a future course of action. (Use the 
blank sheet following test set “A".) 

Answer : a) Your graph should have an axis for number of students and 
one for number of or percent of correct scores, 
b) Your memo should contain the following points: 

1) all of the students met the objective. 

2) there was an average gain of 5 points on post-testing. 

3) this appears an appropriate objective for a similar 
target population in the future. 

4) the instruction appears effective and should be re- 
tained unless there are unidentified (in this trainee 
test) constraints. 
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You have prepared some instructional material for use in a 
high school class. Nov/ you are ready to try out the material v/ith 
one student to see if the material v/orks. 

In each of the conversations belov/, place a check beside the 
teacher response you feel is most appropriate. 



STUDEHT: 


Is this the French test you v/ant me to take? 


P} 1st TLACHER'S 
RESPONSE: 


Yes, try to do a careful job. Read all the 
materials and answer all of the questions. 


□ 2nd TEACHER'S 
RESPONSE: 


This is some instructional material which has 
been developed for a French class. You are 
going to help us find out how well it works. 


STUDENT: 


Shall I start working on this now? 


□ 1st TEACHER’S 
RESPONSE: 


Yes, we're ready. You can begin any time. 


□ 2nd TEACHER'S 
RESPONSE: 


First I would like to find out how much of 
this material you knov/. 


STUDENT; 


O.K. Now I'll start on number 12. (Long pause 
and look of bewilderment) - (gives up and 
scribbles an answer in desperation.) 


□ 1st TEACHER'S 
RESPONSE: 


(before student gives up and scribbles answer) 
Nhat is the problem? 


□ 2nd TEACHER’S 
RESPONSE; 


(after student gives up and scribbles answer) 
Now number 13. 


STUDENT: 


I'm confused by this question. Do I circle the 
right ansv/er or what? 


□ 1st TEACHER'S 
RESPONSE: 


Would it help if I wrote these instructions? 


□ 2nd TEACHER’S 
RESPONSE: 


Mo, match this answer with a sentence in this 
paragraph. 
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STUDENT: (looks puzzled) I prqbabiy should understand 

this, but this rule doesn't make sense. 



□ 1st TEACHER'S 

RESPONSE: Remember your first semester course? It prob- 
ably was mentioned then. 

n 2nd TEACHER'S 

RESPOflSE: Can you tell me why the rule Is confusing? 



STUDENT: I can't do this; I don't know what placebo 
means . 



□ 1st TEACHER'S 

RESPONSE; Would It help If I Inserted this Information 
about what a placebo is « . .? 

n 2nd TEACHER'S 

RESPONSE: Don't worry about that - go on to the next 
paragraph. 



